feat: add server conformance tests for SEP-2575#271
Conversation
pcarleton
left a comment
There was a problem hiding this comment.
Thanks for this — covers a good chunk of the version-negotiation and discover surface. (reminder the full SEP-2575 traceability YAML in #273 as a shared reference for this and #270.)
Coverage: the YAML has 20 server-side check: rows; this PR covers 7 (discover, version-error, 400-on-unsupported, header-mismatch, 404-on-unknown, missing-capability ×2). The big uncovered areas are subscriptions/listen (4 rows), HTTP cancellation (3 rows - 2 HTTP, 1 stdio), the core statelessness invariants (no-prior-context / no-connection-reuse), and server-no-log-without-loglevel. Fine to land those as follow-ups, but worth flagging which are intentionally deferred.
Correctness:
- MUST→WARNING severity —
stateless-error-missing-capabilityandstateless-http-missing-capabilityuseWARNINGon failure, but the spec text is MUST for both-32003and HTTP 400. If the worry is the test tool not existing, skip the check rather than downgrade. stateless-version-response-headeris dead — only ever pushed in acatchblock (stateless.ts:360), never in the happy path. Also no spec backing: there's no server-MUST-return-header. Suggest dropping it.- Registered twice in
index.ts(bothpendingClientScenariosListandallClientScenariosList). We should clean this up, but you can put it inalland we filter it by spec version tag. - "Absent header" test isn't absent —
{'MCP-Protocol-Version': ''}sets an empty-string header; the helper always injects the default, so the header is never truly omitted. Needs a delete-key path insendRpc. stateless-version-header-mismatchonly checks HTTP 400; spec says it SHOULD include-32001 HeaderMismatchin the body.stateless-removed-*(5 checks) — there's no normative MUST forbiddingping/initialize; the only relevant text is the generic "unknown method → 404 + -32601". These checks don't assert the 404, and the example returns 200 for them. Suggest folding into onesep-2575-http-server-method-not-found-404slug parametrized by method.stateless-meta-missing*(4 checks) — the draft spec pages only put the MUST on the client (the SEP doc has a server-MUST-reject, but it didn't make it into basic/index.mdx — possibly a spec bug to fix). Either way, one check slug with the field name in details is probably better here.noDriftrequires identical array order between discover and error; spec doesn't mandate order — set comparison.- Check IDs — should be
sep-2575-*to match #273 (same note left on #270);specReferencesshould point to spec anchors not the SEP PR.
Structure: ~60% of the 749 lines is duplicated catch-block boilerplate re-pushing id/name/description. A runCheck(id, desc, fn) helper would cut it to ~250 lines and prevent the try/catch description drift (e.g. stateless-version-unsupported has different descriptions in try vs catch).
Negative test could also be tightened — expect(failures.length).toBeGreaterThan(0) doesn't pin which checks fail; #270 uses expectedFailureSlugs which is the pattern we want.
Add conformance tests for SEP-2575 to validate the stateless-MCP behavior.
Motivation and Context
Ensure that servers adhere to the SEP.
How Has This Been Tested?
Tested locally
Breaking Changes
n/a. This tests are for the draft specs.
Types of changes
Checklist
server conformance tests for #266